CoBeL-RL: A neuroscience-oriented simulation framework for complex behavior and learning.

Reinforcement learning (RL) has become a popular paradigm for modeling animal behavior, analyzing neuronal representations, and studying their emergence during learning. This development has been fueled by advances in understanding the role of RL in both the brain and artificial intelligence. However, while in machine learning a set of tools and standardized benchmarks facilitate the development of new methods and their comparison to existing ones, in neuroscience, the software infrastructure is much more fragmented. Even if sharing theoretical principles, computational studies rarely share software frameworks, thereby impeding the integration or comparison of different results. Machine learning tools are also difficult to port to computational neuroscience since the experimental requirements are usually not well aligned. To address these challenges we introduce CoBeL-RL, a closed-loop simulator of complex behavior and learning based on RL and deep neural networks. It provides a neuroscience-oriented framework for efficiently setting up and running simulations. CoBeL-RL offers a set of virtual environments, e.g., T-maze and Morris water maze, which can be simulated at different levels of abstraction, e.g., a simple gridworld or a 3D environment with complex visual stimuli, and set up using intuitive GUI tools. A range of RL algorithms, e.g., Dyna-Q and deep Q-network algorithms, is provided and can be easily extended. CoBeL-RL provides tools for monitoring and analyzing behavior and unit activity, and allows for fine-grained control of the simulation via interfaces to relevant points in its closed-loop. In summary, CoBeL-RL fills an important gap in the software toolbox of computational neuroscience.

Differential Hebbian learning with time-continuous signals for active noise reduction.

Spike timing-dependent plasticity, related to differential Hebb-rules, has become a leading paradigm in neuronal learning, because weights can grow or shrink depending on the timing of pre- and post-synaptic signals. Here we use this paradigm to reduce unwanted (acoustic) noise. Our system relies on heterosynaptic differential Hebbian learning and we show that it can efficiently eliminate noise by up to -140 dB in multi-microphone setups under various conditions. The system quickly learns, most often within a few seconds, and it is robust with respect to different geometrical microphone configurations, too. Hence, this theoretical study demonstrates that it is possible to successfully transfer differential Hebbian learning, derived from the neurosciences, into a technical domain.

Adaptive Extreme Edge Computing for Wearable Devices.

Wearable devices are a fast-growing technology with impact on personal healthcare for both society and economy. Due to the widespread of sensors in pervasive and distributed networks, power consumption, processing speed, and system adaptation are vital in future smart wearable devices. The visioning and forecasting of how to bring computation to the edge in smart sensors have already begun, with an aspiration to provide adaptive extreme edge computing. Here, we provide a holistic view of hardware and theoretical solutions toward smart wearable devices that can provide guidance to research in this pervasive computing era. We propose various solutions for biologically plausible models for continual learning in neuromorphic computing technologies for wearable sensors. To envision this concept, we provide a systematic outline in which prospective low power and low latency scenarios of wearable sensors in neuromorphic platforms are expected. We successively describe vital potential landscapes of neuromorphic processors exploiting complementary metal-oxide semiconductors (CMOS) and emerging memory technologies (e.g., memristive devices). Furthermore, we evaluate the requirements for edge computing within wearable devices in terms of footprint, power consumption, latency, and data size. We additionally investigate the challenges beyond neuromorphic computing hardware, algorithms and devices that could impede enhancement of adaptive edge computing in smart wearable devices.

Structural plasticity on an accelerated analog neuromorphic hardware system.

In computational neuroscience, as well as in machine learning, neuromorphic devices promise an accelerated and scalable alternative to neural network simulations. Their neural connectivity and synaptic capacity depend on their specific design choices, but is always intrinsically limited. Here, we present a strategy to achieve structural plasticity that optimizes resource allocation under these constraints by constantly rewiring the pre- and postsynaptic partners while keeping the neuronal fan-in constant and the connectome sparse. In particular, we implemented this algorithm on the analog neuromorphic system BrainScaleS-2. It was executed on a custom embedded digital processor located on chip, accompanying the mixed-signal substrate of spiking neurons and synapse circuits. We evaluated our implementation in a simple supervised learning scenario, showing its ability to optimize the network topology with respect to the nature of its training data, as well as its overall computational efficiency.

Cometary Comae-Surface Links: The Physics of Gas and Dust from the Surface to a Spacecraft.

A comet is a highly dynamic object, undergoing a permanent state of change. These changes have to be carefully classified and considered according to their intrinsic temporal and spatial scales. The Rosetta mission has, through its contiguous in-situ and remote sensing coverage of comet 67P/Churyumov-Gerasimenko (hereafter 67P) over the time span of August 2014 to September 2016, monitored the emergence, culmination, and winding down of the gas and dust comae. This provided an unprecedented data set and has spurred a large effort to connect in-situ and remote sensing measurements to the surface. In this review, we address our current understanding of cometary activity and the challenges involved when linking comae data to the surface. We give the current state of research by describing what we know about the physical processes involved from the surface to a few tens of kilometres above it with respect to the gas and dust emission from cometary nuclei. Further, we describe how complex multidimensional cometary gas and dust models have developed from the Halley encounter of 1986 to today. This includes the study of inhomogeneous outgassing and determination of the gas and dust production rates. Additionally, the different approaches used and results obtained to link coma data to the surface will be discussed. We discuss forward and inversion models and we describe the limitations of the respective approaches. The current literature suggests that there does not seem to be a single uniform process behind cometary activity. Rather, activity seems to be the consequence of a variety of erosion processes, including the sublimation of both water ice and more volatile material, but possibly also more exotic processes such as fracture and cliff erosion under thermal and mechanical stress, sub-surface heat storage, and a complex interplay of these processes. Seasons and the nucleus shape are key factors for the distribution and temporal evolution of activity and imply that the heliocentric evolution of activity can be highly individual for every comet, and generalisations can be misleading.

Optimal Resources for Rural Surgery.

BACKGROUND: Timely access to emergency general surgery services, including trauma, is a critical aspect of patient care. This study looks to identify resource availability at small rural hospitals in order to improve the quality of surgical care. METHODS: Forty-five nonteaching hospitals in West Virginia were divided into large community hospitals with multiple specialties (LCHs), small community hospitals with fewer specialties (SCHs), and critical access hospitals (CAHs). A 58-question survey on optimal resources for surgery was completed by 1 representative surgeon at each hospital. There were 8 LCHs, 18 SCHs, and 19 CAHs with survey response rates of 100%, 83%, and 89%, respectively. RESULTS: One hundred percent of hospitals surveyed had respiratory therapy and ventilator support, computerized tomography (CT) scanner and ultrasound, certified operating rooms, lab support, packed red blood cells (PRBC), and FFP accessible 24/7. Availability of cryoprecipitate, platelets, tranexamic acid (TXA), and prothrombin complex concentrate (PCC) decreased from LCHs to CAHs. The majority had board-certified general surgeons; however, only 86% LCHs, 53% SCHs, and 50% CAHs had advanced trauma life support (ATLS) certification. One hundred percent of LCHs had operating room (OR) crew on call within 30 minutes, emergency cardiovascular equipment, critical care nursing, on-site pathologist, and biologic/synthetic mesh, whereas fewer SCHs and CAHs had these resources. One hundred percent of LCHs and SCHs had anesthesia availability 24/7 compared to 78% of CAHs. DISCUSSION: Improving access to the aforementioned resources is of utmost importance to patient outcomes. This will enhance rural surgical care and decrease emergency surgical transfers. Further education and research are necessary to support and improve rural trauma systems.

Ammonium salts are a reservoir of nitrogen on a cometary nucleus and possibly on some asteroids.

The measured nitrogen-to-carbon ratio in comets is lower than for the Sun, a discrepancy which could be alleviated if there is an unknown reservoir of nitrogen in comets. The nucleus of comet 67P/Churyumov-Gerasimenko exhibits an unidentified broad spectral reflectance feature around 3.2 micrometers, which is ubiquitous across its surface. On the basis of laboratory experiments, we attribute this absorption band to ammonium salts mixed with dust on the surface. The depth of the band indicates that semivolatile ammonium salts are a substantial reservoir of nitrogen in the comet, potentially dominating over refractory organic matter and more volatile species. Similar absorption features appear in the spectra of some asteroids, implying a compositional link between asteroids, comets, and the parent interstellar cloud.

Autism and Advance Directives: Determining Capability and the Use of Health-Care Tools to Aid in Effective Communication and Decision-Making.

With the growing number of individuals with Autism Spectrum Disorder (ASD) reaching the age of consent, health-care providers must be prepared to bridge gaps in their knowledge of ASD. This is especially true for clinicians who may have to determine if a person with ASD has the capacity to engage in end-of-life decision making, complete advance directives, or act as a surrogate decision maker for someone else. This paper provides an overview of the unique characteristics of autism as related to the communication, cognitive processing, and the capability to participate in advance care planning and, when acting as a surrogate decision maker, to consider the values and preferences of others. In addition, we examine the roles and responsibilities of clinician as facilitator of shared health-care decision making communication with the individual who has autism. Consideration is given to determining capacity, planning for atypical responses, the impact or lack of influence of the framing effect, and strategies for presenting information. Finally, we will offer health-care providers information and examples for adapting their existing end-of-life decision-making tools and conversation guides to meet the communication needs of persons with ASD.

Embodied Synaptic Plasticity With Online Reinforcement Learning.

The endeavor to understand the brain involves multiple collaborating research fields. Classically, synaptic plasticity rules derived by theoretical neuroscientists are evaluated in isolation on pattern classification tasks. This contrasts with the biological brain which purpose is to control a body in closed-loop. This paper contributes to bringing the fields of computational neuroscience and robotics closer together by integrating open-source software components from these two fields. The resulting framework allows to evaluate the validity of biologically-plausibe plasticity models in closed-loop robotics environments. We demonstrate this framework to evaluate Synaptic Plasticity with Online REinforcement learning (SPORE), a reward-learning rule based on synaptic sampling, on two visuomotor tasks: reaching and lane following. We show that SPORE is capable of learning to perform policies within the course of simulated hours for both tasks. Provisional parameter explorations indicate that the learning rate and the temperature driving the stochastic processes that govern synaptic learning dynamics need to be regulated for performance improvements to be retained. We conclude by discussing the recent deep reinforcement learning techniques which would be beneficial to increase the functionality of SPORE on visuomotor tasks.

Disaster Ethics: Shifting Priorities in an Unstable and Dangerous Environment.

Emergency and critical care medicine are fraught with ethically challenging decision making for clinicians, patients, and families. Time and resource constraints, decisional-impaired patients, and emotionally overwhelmed family members make obtaining informed consent, discussing withholding or withdrawing of life-sustaining treatments, and respecting patient values and preferences difficult. When illness or trauma is secondary to disaster, ethical considerations increase and change based on number of casualties, type of disaster, and anticipated life cycle of the crisis. This article considers the ethical issues that arise when health providers are confronted with the challenges of caring for victims of disaster.

Efficient Reward-Based Structural Plasticity on a SpiNNaker 2 Prototype.

Advances in neuroscience uncover the mechanisms employed by the brain to efficiently solve complex learning tasks with very limited resources. However, the efficiency is often lost when one tries to port these findings to a silicon substrate, since brain-inspired algorithms often make extensive use of complex functions, such as random number generators, that are expensive to compute on standard general purpose hardware. The prototype chip of the second generation SpiNNaker system is designed to overcome this problem. Low-power advanced RISC machine (ARM) processors equipped with a random number generator and an exponential function accelerator enable the efficient execution of brain-inspired algorithms. We implement the recently introduced reward-based synaptic sampling model that employs structural plasticity to learn a function or task. The numerical simulation of the model requires to update the synapse variables in each time step including an explorative random term. To the best of our knowledge, this is the most complex synapse model implemented so far on the SpiNNaker system. By making efficient use of the hardware accelerators and numerical optimizations, the computation time of one plasticity update is reduced by a factor of 2. This, combined with fitting the model into to the local static random access memory (SRAM), leads to 62% energy reduction compared to the case without accelerators and the use of external dynamic random access memory (DRAM). The model implementation is integrated into the SpiNNaker software framework allowing for scalability onto larger systems. The hardware-software system presented in this paper paves the way for power-efficient mobile and biomedical applications with biologically plausible brain-inspired algorithms.

Memory-Efficient Deep Learning on a SpiNNaker 2 Prototype.

The memory requirement of deep learning algorithms is considered incompatible with the memory restriction of energy-efficient hardware. A low memory footprint can be achieved by pruning obsolete connections or reducing the precision of connection strengths after the network has been trained. Yet, these techniques are not applicable to the case when neural networks have to be trained directly on hardware due to the hard memory constraints. Deep Rewiring (DEEP R) is a training algorithm which continuously rewires the network while preserving very sparse connectivity all along the training procedure. We apply DEEP R to a deep neural network implementation on a prototype chip of the 2nd generation SpiNNaker system. The local memory of a single core on this chip is limited to 64 KB and a deep network architecture is trained entirely within this constraint without the use of external memory. Throughout training, the proportion of active connections is limited to 1.3%. On the handwritten digits dataset MNIST, this extremely sparse network achieves 96.6% classification accuracy at convergence. Utilizing the multi-processor feature of the SpiNNaker system, we found very good scaling in terms of computation time, per-core memory consumption, and energy constraints. When compared to a X86 CPU implementation, neural network training on the SpiNNaker 2 prototype improves power and energy consumption by two orders of magnitude.

A Dynamic Connectome Supports the Emergence of Stable Computational Function of Neural Circuits through Reward-Based Learning.

Synaptic connections between neurons in the brain are dynamic because of continuously ongoing spine dynamics, axonal sprouting, and other processes. In fact, it was recently shown that the spontaneous synapse-autonomous component of spine dynamics is at least as large as the component that depends on the history of pre- and postsynaptic neural activity. These data are inconsistent with common models for network plasticity and raise the following questions: how can neural circuits maintain a stable computational function in spite of these continuously ongoing processes, and what could be functional uses of these ongoing processes? Here, we present a rigorous theoretical framework for these seemingly stochastic spine dynamics and rewiring processes in the context of reward-based learning tasks. We show that spontaneous synapse-autonomous processes, in combination with reward signals such as dopamine, can explain the capability of networks of neurons in the brain to configure themselves for specific computational tasks, and to compensate automatically for later changes in the network or task. Furthermore, we show theoretically and through computer simulations that stable computational performance is compatible with continuously ongoing synapse-autonomous changes. After reaching good computational performance it causes primarily a slow drift of network architecture and dynamics in task-irrelevant dimensions, as observed for neural activity in motor cortex and other areas. On the more abstract level of reinforcement learning the resulting model gives rise to an understanding of reward-driven network plasticity as continuous sampling of network configurations.

Recurrent Spiking Networks Solve Planning Tasks.

A recurrent spiking neural network is proposed that implements planning as probabilistic inference for finite and infinite horizon tasks. The architecture splits this problem into two parts: The stochastic transient firing of the network embodies the dynamics of the planning task. With appropriate injected input this dynamics is shaped to generate high-reward state trajectories. A general class of reward-modulated plasticity rules for these afferent synapses is presented. The updates optimize the likelihood of getting a reward through a variant of an Expectation Maximization algorithm and learning is guaranteed to convergence to a local maximum. We find that the network dynamics are qualitatively similar to transient firing patterns during planning and foraging in the hippocampus of awake behaving rats. The model extends classical attractor models and provides a testable prediction on identifying modulating contextual information. In a real robot arm reaching and obstacle avoidance task the ability to represent multiple task solutions is investigated. The neural planning method with its local update rules provides the basis for future neuromorphic hardware implementations with promising potentials like large data processing abilities and early initiation of strategies to avoid dangerous situations in robot co-worker scenarios.

Network Plasticity as Bayesian Inference.

General results from statistical learning theory suggest to understand not only brain computations, but also brain plasticity as probabilistic inference. But a model for that has been missing. We propose that inherently stochastic features of synaptic plasticity and spine motility enable cortical networks of neurons to carry out probabilistic inference by sampling from a posterior distribution of network configurations. This model provides a viable alternative to existing models that propose convergence of parameters to maximum likelihood values. It explains how priors on weight distributions and connection probabilities can be merged optimally with learned experience, how cortical networks can generalize learned information so well to novel experiences, and how they can compensate continuously for unforeseen disturbances of the network. The resulting new theory of network plasticity explains from a functional perspective a number of experimental data on stochastic aspects of synaptic plasticity that previously appeared to be quite puzzling.

NEVESIM: event-driven neural simulation framework with a Python interface.

NEVESIM is a software package for event-driven simulation of networks of spiking neurons with a fast simulation core in C++, and a scripting user interface in the Python programming language. It supports simulation of heterogeneous networks with different types of neurons and synapses, and can be easily extended by the user with new neuron and synapse types. To enable heterogeneous networks and extensibility, NEVESIM is designed to decouple the simulation logic of communicating events (spikes) between the neurons at a network level from the implementation of the internal dynamics of individual neurons. In this paper we will present the simulation framework of NEVESIM, its concepts and features, as well as some aspects of the object-oriented design approaches and simulation strategies that were utilized to efficiently implement the concepts and functionalities of the framework. We will also give an overview of the Python user interface, its basic commands and constructs, and also discuss the benefits of integrating NEVESIM with Python. One of the valuable capabilities of the simulator is to simulate exactly and efficiently networks of stochastic spiking neurons from the recently developed theoretical framework of neural sampling. This functionality was implemented as an extension on top of the basic NEVESIM framework. Altogether, the intended purpose of the NEVESIM framework is to provide a basis for further extensions that support simulation of various neural network models incorporating different neuron and synapse types that can potentially also use different simulation strategies.

STDP installs in Winner-Take-All circuits an online approximation to hidden Markov model learning.

In order to cross a street without being run over, we need to be able to extract very fast hidden causes of dynamically changing multi-modal sensory stimuli, and to predict their future evolution. We show here that a generic cortical microcircuit motif, pyramidal cells with lateral excitation and inhibition, provides the basis for this difficult but all-important information processing capability. This capability emerges in the presence of noise automatically through effects of STDP on connections between pyramidal cells in Winner-Take-All circuits with lateral excitation. In fact, one can show that these motifs endow cortical microcircuits with functional properties of a hidden Markov model, a generic model for solving such tasks through probabilistic inference. Whereas in engineering applications this model is adapted to specific tasks through offline learning, we show here that a major portion of the functionality of hidden Markov models arises already from online applications of STDP, without any supervision or rewards. We demonstrate the emergent computing capabilities of the model through several computer simulations. The full power of hidden Markov model learning can be attained through reward-gated STDP. This is due to the fact that these mechanisms enable a rejection sampling approximation to theoretically optimal learning. We investigate the possible performance gain that can be achieved with this more accurate learning method for an artificial grammar task.